Skip to content

computer: run several actions in one call - #2

Open
DroBov1Tya wants to merge 1 commit into
mainfrom
computer-action-batch
Open

DroBov1Tya wants to merge 1 commit into
mainfrom
computer-action-batch

Conversation

@DroBov1Tya

Copy link
Copy Markdown
Owner

Adds computer_actions: up to 20 click, type_text, press_keys, scroll and screenshot steps in one call, so the model spends fewer round trips and tokens when it already knows where to act.

A batch never gets less scrutiny than its steps one by one (two independent security reviews):

  • every step is checked before anything is asked; mode switches and tool-scoped before_tool hooks apply per step; the arbiter judges each step alone;
  • the approval carries every step, numbered as run, covered ones marked; the terminal pages and refuses a yes until every step was shown (plus 600 ms against a held Enter); desktop and browser list them;
  • the request's own text lists the steps too, so messaging platforms see them; a request too long to show whole in a chat can only be denied there;
  • press_keys refuses control characters; PROTOCOL_VERSION 10 -> 11 for the new approval field.

Not verified: a live batch on macOS or Linux desktops, a Linux build, how a 20-step approval looks in each client and in Telegram.

A known sequence (click a field, type, press enter, look) cost one model
round trip per action. computer_actions runs up to 20 click, type_text,
press_keys, scroll and screenshot steps in order in one call.

A batch must never get less scrutiny than its steps one by one:
- every step is checked before anything is asked or done;
- a step whose tool is switched off in the mode, or that a before_tool
  hook denies (run per step with its own tool and arguments), refuses
  the whole batch;
- one approval lists every step by its batch number, steps a session
  grant covers marked; mixed kinds get no session grant; the arbiter
  judges each step alone;
- the steps travel structured in ApprovalRequest (protocol 11) and in
  the request text, so every client and chat platform shows each one;
  the terminal pages long batches and lets a yes count only after all
  steps were shown. A chat platform that would cut a long request gets
  no yes for it: it can only deny there, the app allows it.

The single tools now build the same checked step as the batch, so the
two cannot ask or act differently. Every click in a batch reads the
picture taken before it. press_keys refuses control characters and
shows keys by their normal names.

Not verified:
- a live batch on a real macOS desktop (AXPress clicks, typing, keys,
  screenshots inside a batch, coordinates after an inner screenshot);
- the same on Linux X11 and Wayland, and a Linux build;
- the approval drawn in the terminal, desktop and browser clients and
  in Telegram with a 20-step batch.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant